Automatic Evaluation of Summary Using Textual Entailment

This paper describes about an automatic technique of evaluating summary. The standard and popular summary evaluation techniques or tools are not fully automatic; they all need some manual process. Using textual entailment (TE) the generated summary can be evaluated automatically without any manual evaluation/process. The TE system is the composition of lexical entailment module, lexical distance module, Chunk module, Named Entity module and syntactic text entailment (TE) module. The syntactic TE system is based on the Support Vector Machine (SVM) that uses twenty five features for lexical similarity, the output tag from a rule based syntactic two-way TE system as a feature and the outputs from a rule based Chunk Module and Named Entity Module as the other features. The documents are used as text (T) and summary of these documents are taken as hypothesis (H). So, if the information of documents is entailed into the summary then it will be a very good summary. After comparing with the ROUGE 1.5.5 evaluation scores, the proposed evaluation technique achieved a high accuracy of 98.25% w.r.t ROUGE-2 and 95.65% w.r.t ROUGE-SU4.